Replicating and Scaling up Qualitative Analysis using Crowdsourcing: A Github-based Case Study

Due to the diculties in replicating and scaling up qualitative studies, such studies are rarely veri€ed. Accordingly, in this paper, we leverage the advantages of crowdsourcing (low costs, fast speed, scalable workforce) to replicate and scale-up one state-of-the-art qualitative study. Œat qualitative study explored 20 GitHub pull requests to learn factors that inƒuence the fate of pull requests with respect to approval and merging. As a secondary study, using crowdsourcing at a cost of $200, we studied 250 pull requests from 142 GitHub projects. Œe prior qualitative €ndings are mapped into questions for crowds workers. Œeir answers were converted into binary features to build a predictor which predicts whether code would be merged with median F1 scores of 68%. For the same large group of pull requests, the median F1 scores could achieve 90% by a predictor built with additional features de€ned by prior quantitative results. Based on this case study, we conclude that there is much bene€t in combining di‚erent kinds of research methods. While qualitative insights are very useful for €nding novel insights, they can be hard to scale or replicate. Œat said, they can guide and de€ne the goals of scalable secondary studies that use (e.g.) crowdsourcing+data mining. On the other hand, while data mining methods are reproducible and scalable to large data sets, their results may be spectacularly wrong since they lack contextual information. Œat said, they can be used to test the stability and external validity, of the insights gained from a qualitative analysis.